Imaging apparatus, imaging method, recording medium, and program

ABSTRACT

Disclosed herein are an imaging apparatus, an imaging method, a recording medium, and a program which are capable of processing imaging data in a manner similar to that of a normal frame rate. Image data captured by a solid-state imaging element capable of performing imaging at a high resolution and a high frame rate is supplied to a memory control. The memory control, at the same time as writing the imaging data input from the imaging element in a frame memory, reads preceding frames of imaging data that are recorded on the frame memory, and sequentially respectively outputs them in parallel, as video image data items, for each frame, to respective camera signal processing units. In the camera signal processing units, a video output, a viewfinder output, codec units, and recording units, processing similar to that when a frame rate that is ¼ the imaging frame rate is executed.

CROSS REFERENCE TO RELATED APPLICATIONS

This document is a Continuation application of, is based upon and claimsthe benefit of Priority under 35 §U.S.C. 119 from U.S. Ser. No.12/663,538, filed Dec. 8, 2009, herein incorporated by reference, whichis a National Stage application of International Application No.PCT/JP2008/061835, filed Jun. 30, 2008, which is based upon and claimsthe benefit of priority form prior Japanese Patent Application No.2007-171672, filed Jun. 29, 2007.

TECHNICAL FIELD

The present invention relates to an imaging apparatus, an imagingmethod, a recording medium, and a program. More specifically, thepresent invention relates to an imaging apparatus, an imaging method, arecording medium, and a program which are suitable for use in capturinga moving image at a high frame rate.

BACKGROUND ART

In recent years, high-speed imaging apparatuses capable of performingimaging at a speed higher than a normal video frame rate (60 frames persecond, 50 frames per second, 24 frames per second, or the like) havebecome widely used.

For realization of imaging and recording at a high frame rate, forexample, high-speed imaging apparatuses capable of realizing high-speedimaging with a reduced number of pixels that are read from solid-stateimaging elements within one frame without increasing the speed of thesubsequent processing have been available. Such high-speed imagingapparatuses employ a technique for recording a plurality of frame imageswith a reduced number of images within one frame of a standard videosignal in such a manner that the plurality of frame images are joinedtogether (see, for example, Patent Document 1) or a technique forperforming recording onto a semiconductor memory using a dedicatedcompression scheme or image format (see, for example, Patent Document2).

-   [Patent Document 1] Japanese Unexamined Patent Application    Publication No. 8-88833-   [Patent Document 2] Japanese Unexamined Patent Application    Publication No. 2006-319513

Additionally, high-speed imaging apparatuses in which imaging dataoutput by driving a solid-state imaging element at a high speed isrecorded directly onto a semiconductor memory to realize high-speedimaging have been available. In many of such high-speed imagingapparatuses, since imaging data is output from a solid-state imagingelement at a speed that is too high to perform subsequent signalprocessing, uncompressed RAW data is recorded without any change. Suchhigh-speed imaging apparatuses have been commercially available mainlyfor industrial inspection.

Then, additionally, high-speed imaging apparatuses that realizehigh-speed imaging by spatially dividing an image of one frame andprocessing individual areas in parallel have been available. Suchhigh-speed imaging apparatuses employ a technique for distributing anoutput from a solid-state imaging element in units of horizontal linesand performing parallel processing (see, for example, Patent Document 3)or a technique for splitting incident light using a prism, supplyingresulting light components to a plurality of solid-state imagingelements, and processing output signals of these solid-state imagingelements in parallel (see, for example, Patent Document 4).

-   [Patent Document 3] Japanese Unexamined Patent Application    Publication No. 1-286586-   [Patent Document 4] Japanese Unexamined Patent Application    Publication No. 5-316402

DISCLOSURE OF INVENTION Technical Problem

However, in a case where the number of pixels read from a solid-stateimaging element is reduced, it is not possible to obtain a high spatialresolution. Additionally, recording of an image in which a plurality offrames are bonded to one another requires image conversion processingduring reproduction, resulting in increased complexity of theprocessing.

Additionally, in a case where imaging data output by driving asolid-state imaging element at a high speed is recorded directly onto asemiconductor memory to realize high-speed imaging, it is possible toachieve imaging at a high resolution and a high frame rate althoughimaging data is output from the solid-state imaging element at a speedthat is so high as to require the use of a semiconductor memory as arecording device. However, due to a limit in the capacity of asemiconductor memory that can be mounted in an imaging apparatus,long-time recording is difficult.

Additionally, if an image is spatially divided into segments and thesegments are processed in parallel in order to perform image processingat a high speed, it is possible to record a high-resolution andhigh-frame-rate image for a long time. However, since each recordedimage data segment is an image divided horizontally or vertically in theshape of a strip or the shape of a rectangular area, it is necessary toperform a process of combining these images during reproduction,resulting in increased complexity of the processing. Additionally, sinceeach recorded image segment is a recorded spatial portion of an actualframe, it has been meaningless to individually reproduce the respectiverecorded image segments.

In other words, each recorded image data segment has been data thatcannot be utilized as it is.

The present invention has been made in view of such situations, and isintended to facilitate easy processing of an image captured at a highresolution and a high frame rate.

Technical Solution

An imaging apparatus of an aspect of the present invention includesimaging means for obtaining imaging data of a first rate; data dividingmeans for distributing the imaging data of the first rate, which iscaptured by the imaging means, in units of frames and dividing theimaging data into N channels of moving image data of a second rate thatis a rate that is 1/N the first rate (where N is a positive integer);and N image processing means for processing in parallel the N channelsof moving image data obtained by the dividing means.

The imaging apparatus can be configured to further include output meansfor outputting the N channels of moving image data processed by theimage processing means, and the output means can be configured to outputonly one channel of the N channels of moving image data or to output aresult obtained by performing frame combination on at least a portion ofthe N channels of moving image data on the basis of a rate of movingimage data to be output.

The imaging apparatus can be configured to further include recordingmeans for recording the N channels of moving image data processed by theimage processing means, and the output means can be configured to outputat least a portion of the N channels of the moving image data recordedon the recording means.

The recording means can be configured such that N recording means areprovided or the recording means is divided into N areas, and can beconfigured to respectively record the N channels of moving image dataprocessed by the image processing means.

The second rate can be configured to be 60 frames per second.

The second rate can be configured to be 50 frames per second.

The second rate can be configured to be 24 frames per second.

The N channels can be configured to be four channels.

The N channels can be configured to be two channels.

The first rate can be configured to be 240 frames per second.

The imaging apparatus can be configured to further include recordingmeans for recording the N channels of moving image data processed by theimage processing means.

The recording means can be configured such that N recording means areprovided or the recording means is divided into N areas, and can beconfigured to respectively record the N channels of moving image dataprocessed by the image processing means.

The imaging apparatus can be configured to further include encodingmeans for encoding the N channels of moving image data processed by theimage processing means, and the recording means can be configured torecord the N channels of moving image data encoded by the encodingmeans.

The imaging apparatus can be configured to further include decodingmeans for decoding the N channels of moving image data encoded by theencoding means and recorded by the recording means; and output means foroutputting the N channels of moving image data decoded by the decodingmeans. The decoding means can be configured to decode only one channelof the N channels of moving image data or to decode at least a portionof the N channels of moving image data on the basis of a rate of movingimage data to be output. The output means can be configured to outputthe one channel of the N channels of moving image data, which is decodedby the decoding means, or to output a result obtained by performingframe combination on at least a portion of the N channels of movingimage data on the basis of a rate of moving image data to be output.

An imaging method of an aspect of the present invention is an imagingmethod for an imaging apparatus that captures moving image data, andincludes the steps of performing imaging at a first rate; dividingcaptured imaging data of the first rate in units of frames into Nchannels of moving image data of a second rate that is a rate that is1/N the first rate (where N is a positive integer); and processing theobtained N channels of moving image data using N parallel units.

A program of an aspect of the present invention is a program for causinga computer to execute a process of capturing moving image data, andcauses the computer to execute a process including the steps ofcontrolling imaging at a first rate; dividing captured imaging data ofthe first rate in units of frames into N channels of moving image dataof a second rate that is a rate that is 1/N the first rate (where N is apositive integer); and processing the obtained N channels of movingimage data using N parallel units.

In an aspect of the present invention, an image is captured at a firstrate, captured imaging data of the first rate is divided in units offrames into N channels of moving image data of a second rate that is arate that is 1/N the first rate (where N is a positive integer), and theobtained N channels of moving image data are processed using N parallelunits.

The term network refers to a mechanism in which at least two apparatusesare connected to each other so that information can be transmitted fromone apparatus to another apparatus. Apparatuses that communicate withone another via a network may be independent apparatuses or internalblocks that constitute a single apparatus.

Additionally, the term communication refers to communication which mayinclude, as well as wireless communication and wired communication,communication including both wireless communication and wiredcommunication, that is, communication where wireless communication isperformed in a certain period while wired communication is performed inanother period. Furthermore, communication from one apparatus to anotherapparatus may be performed via wired communication, and communicationfrom the other apparatus to the one apparatus may be performed viawireless communication.

An imaging apparatus may be an independent apparatus or may be a blockthat performs imaging processing, which is provided in an imageprocessing apparatus, an information processing apparatus, arecording/reproducing apparatus, or the like.

Advantageous Effects

As above, according to an aspect of the present invention, a movingimage can be captured and, in particular, even during imaging at a highframe rate, an image can be processed in a manner similar to that duringimaging at a normal frame rate.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a configuration of an imagingapparatus to which the present invention is applied.

FIG. 2 is a diagram illustrating a Bayer pattern.

FIG. 3 is a diagram for explaining frame distribution.

FIG. 4 is a diagram for explaining frame distribution.

FIG. 5 is a block diagram illustrating a configuration of a camerasignal processing unit of FIG. 1.

FIG. 6 is a diagram for explaining frame combination.

FIG. 7 is a diagram for explaining frame combination.

FIG. 8 is a block diagram illustrating a different configuration of theimaging apparatus to which the present invention is applied.

FIG. 9 is a block diagram illustrating a configuration of a camerasignal processing unit of FIG. 8.

FIG. 10 is a flowchart for explaining an imaging/recording/outputprocess.

FIG. 11 is a flowchart for explaining an imaging data dividing process.

FIG. 12 is a flowchart for explaining the imaging data dividing process.

FIG. 13 is a flowchart for explaining a video output process.

FIG. 14 is a flowchart for explaining the video output process.

FIG. 15 is a diagram illustrating an example of the relationship betweenthe frame rate of a movie and the evaluation value in a case where imagequality was evaluated using five levels.

FIG. 16 is a block diagram illustrating a configuration of a personalcomputer.

EXPLANATION OF REFERENCE NUMERALS

1 imaging apparatus, 11 camera control unit, 21 imaging optical system,22 imaging element, 23 memory control unit, 24 frame memory, 25 camerasignal processing unit, 26 video output unit, 27 viewfinder output unit,28 codec unit, 29 recording unit, 51 WB correction unit, 52 RGBinterpolation synchronization processing unit, 53 matrix processingunit, 54 γ correction unit, 55 color space conversion unit, 101 imagingapparatus, 120 dichroic prism, 121 to 123 imaging element, 124 memorycontrol unit, 125 frame memory, 126 camera signal processing unit, 151WB correction unit

BEST MODE FOR CARRYING OUT THE INVENTION

An embodiment of the present invention will be explained hereinafterwith reference to the drawings.

FIG. 1 is a block diagram illustrating a configuration of an imagingapparatus 1 to which the present invention is applied.

The imaging apparatus 1 is configured to include a camera control unit11, an imaging optical system 21, an imaging element 22, a memorycontrol unit 23, a frame memory 24, a #1 camera signal processing unit25-1, a #2 camera signal processing unit 25-2, a #3 camera signalprocessing unit 25-3, a #4 camera signal processing unit 25-4, a videooutput unit 26, a viewfinder output unit 27, a #1 codec unit 28-1, a #2codec unit 28-2, a #3 codec unit 28-3, a #4 codec unit 28-4, a #1recording unit 29-1, a #2 recording unit 29-2, a #3 recording unit 29-3,and a #4 recording unit 29-4.

The camera control unit 11 is designed to control the overall processingof the imaging apparatus 1.

The imaging apparatus 1 receives, using the imaging element 22, lightincident through the imaging optical system 21 having a mechanicalshutter, a lens, and the like. The imaging element 22 is a solid-stateimaging element capable of performing imaging at a high resolution(here, for example, HD resolution) and a high frame rate (here, forexample, 240 frames per second, that is, referred to as 240 Hz), andoutputs imaging data that is digitally converted via an AD converter(not illustrated), that is, imaging element output data indicated by ain the figures. Here, the AD converter may be mounted on an element in acase where the imaging element 22 is a CMOS solid-state imaging elementor any other case, or may be placed outside the imaging element 22 whenthe imaging element 22 is a solid-state imaging element other than aCMOS solid-state imaging element or any other occasion. Additionally,the imaging element 22 of the imaging apparatus 1 explained using FIG. 1is a single-plate color solid-state imaging element having, on a lightreceiving surface, a color filter that transmits light in differentwavelength ranges for individual pixels.

Since imaging data output from the imaging element 22 is so-called RAWdata that is output from a single-plate color solid-state imagingelement, the imaging data is configured by pixel data having a colorpattern according to a Bayer pattern as illustrated in FIG. 2, and issupplied to the memory control unit 23. Then, in the memory control unit23, a plurality of pixel data items are integrated into units of oneblock that can be exchanged with the frame memory 24 at a time (forexample, a predetermined amount of data into which a plurality of pixeldata items are integrated, such as 64 bits or 128 bits, one line ofdata, or the like), and are stored in the frame memory 24. The memorycontrol unit 23, at the same time as writing the imaging data input fromthe imaging element 22 in the frame memory 24, reads preceding frames ofimaging data from the frame memory 24, and simultaneously outputs, on aframe-by-frame basis, the imaging data as video image data itemsindicated by b to e in the figures to the #1 camera signal processingunit 25-1, the #2 camera signal processing unit 25-2, the #3 camerasignal processing unit 25-3, and the #4 camera signal processing unit25-4.

When the imaging data input from the imaging element 22 to the memorycontrol unit 23, that is, imaging element output data indicated by a inthe figures, is imaging data of HD resolution and 240 frames per second,the video image data items indicated by b to e in the figures, each ofwhich is imaging data of HD resolution and 60 frames per second, areoutput to the #1 camera signal processing unit 25-1, the #2 camerasignal processing unit 25-2, the #3 camera signal processing unit 25-3,and the #4 camera signal processing unit 25-4. Here, the frame memory 24has a capacity to store at least eight frames of imaging data. Thedetails of the operation of the memory control unit 23 will be describedbelow with reference to FIGS. 3 and 4.

The #1 camera signal processing unit 25-1 acquires the imaging data (RAWdata) of HD resolution and 60 frames per second, which is output fromthe memory control unit 23, performs signal processing, and outputs asignal-processed video signal to the #1 codec unit 28-1 and theviewfinder output unit 27. The #2 camera signal processing unit 25-2acquires the imaging data of HD resolution and 60 frames per second,which is output from the memory control unit 23, performs signalprocessing, and outputs a signal-processed video signal to the #2 codecunit 28-2. The #3 camera signal processing unit 25-3 acquires theimaging data of HD resolution and 60 frames per second, which is outputfrom the memory control unit 23, performs signal processing, and outputsa signal-processed video signal to the #3 codec unit 28-3. The #4 camerasignal processing unit 25-4 acquires the imaging data of HD resolutionand 60 frames per second, which is output from the memory control unit23, performs signal processing, and outputs a signal-processed videosignal to the #4 codec unit 28-4.

In the following explanation, each of the #1 camera signal processingunit 25-1, the #2 camera signal processing unit 25-2, the #3 camerasignal processing unit 25-3, and the #4 camera signal processing unit25-4 is referred to simply as a camera signal processing unit 25 unlessthey need to be individually identified. Since each of the camera signalprocessing units 25 processes general imaging data of 60 frames persecond, the imaging apparatus 1 does not require the use of a high-speedsignal processing unit having the capabilities to process imaging dataof 240 frames per second. A further detailed configuration of the camerasignal processing units 25 will be described below with reference toFIG. 5.

The #1 codec unit 28-1 acquires the video signal output from the #1camera signal processing unit 25-1, and performs image encodingprocessing. The #2 codec unit 28-2 acquires the video signal output fromthe #2 camera signal processing unit 25-2, and performs image encodingprocessing. The #3 codec unit 28-3 acquires the video signal output fromthe #3 camera signal processing unit 25-3, and performs image encodingprocessing. The #4 codec unit 28-4 acquires the video signal output fromthe #4 camera signal processing unit 25-4, and performs image encodingprocessing.

The #1 codec unit 28-1 outputs an image-encoded image data stream to the#1 recording unit 29-1. The #2 codec unit 28-2 outputs an image-encodedimage data stream to the #2 recording unit 29-2. The #3 codec unit 28-3outputs an image-encoded image data stream to the #3 recording unit29-3. The #4 codec unit 28-4 outputs an image-encoded image data streamto the #4 recording unit 29-4.

Further, the #1 codec unit 28-1 acquires a compression encoded imagedata stream from the #1 recording unit 29-1, decodes it, and outputsdecoded video image data, which is indicated by f in the figures, to thevideo output unit 26. Similarly, the #1 codec unit 28-1 also outputs thedecoded video image data to the viewfinder output unit 27. The #2 codecunit 28-2 acquires a compression encoded image data stream from the #2recording unit 29-2, decodes it, and outputs decoded video image data,which is indicated by g in the figures, to the video output unit 26. The#3 codec unit 28-3 acquires a compression encoded image data stream fromthe #3 recording unit 29-3, decodes it, and outputs decoded video imagedata, which is indicated by h in the figures, to the video output unit26. The #4 codec unit 28-4 acquires a compression encoded image datastream from the #4 recording unit 29-4, decodes it, and outputs decodedvideo image data, which is indicated by i in the figures, to the videooutput unit 26.

Here, the image encoding processing executed by the #1 codec unit 28-1,the #2 codec unit 28-2, the #3 codec unit 28-3, and the #4 codec unit28-4 is implemented using JPEG 2000 CODEC for intra-frame compression,MPEG 2 or H.264 CODEC for inter-frame compression, or the like. In thefollowing explanation, each of the #1 codec unit 28-1, the #2 codec unit28-2, the #3 codec unit 28-3, and the #4 codec unit 28-4 is referred tosimply as a codec unit 28 unless they need to be individuallyidentified. Since each of the codec units 28 processes general data of60 frames per second, the imaging apparatus 1 does not require the useof a high-speed codec unit having the capabilities to process data of240 frames per second. Additionally, here, explanation has been givenassuming that the codec units 28 are designed to perform encoding anddecoding. However, it goes without saying that, in place of the codecunits 28, decoders and encoders may be individually provided.

The #1 recording unit 29-1, the #2 recording unit 29-2, the #3 recordingunit 29-3, and the #4 recording unit 29-4 acquire and record thecompression encoded image data streams of 60 frames per second, whichare respectively output from the codec units 28. That is, each of the #1recording unit 29-1, the #2 recording unit 29-2, the #3 recording unit29-3, and the #4 recording unit 29-4 records a compression encoded imagedata stream of a video signal of HD resolution and 60 frames per second.

In FIG. 1, each of the #1 recording unit 29-1, the #2 recording unit29-2, the #3 recording unit 29-3, and the #4 recording unit 29-4 isdescribed as an independent recording unit. However, the #1 recordingunit 29-1, the #2 recording unit 29-2, the #3 recording unit 29-3, andthe #4 recording unit 29-4 may be designed to represent differentstorage areas in a single recording unit. In the following explanation,each of the #1 recording unit 29-1, the #2 recording unit 29-2, the #3recording unit 29-3, and the #4 recording unit 29-4 is referred tosimply as a recording unit 29 unless they need to be individuallyidentified. Each of the recording units 29 may be implemented using, forexample, a drive in which a removable medium such as an optical disk isplaced, a hard disk, a semiconductor memory, or the like.

The viewfinder output unit 27 receives the video signal output from the#1 camera signal processing unit 25-1 or the #1 codec unit 28-1, andconverts the video signal into a signal that can be displayed on aviewfinder. The viewfinder, which is not illustrated, is a display uniton which an image that is being photographed or an image that is beingreproduced and output is confirmed, and may be built in or providedoutside the imaging apparatus 1. The viewfinder is configured by, forexample, a liquid crystal display device that displays a video signal(YCbCr 4:2:2). The viewfinder often has a resolution lower than thenormal HD resolution and, additionally, similarly, also has a low framerate. Thus, the viewfinder output unit 27 performs resolution conversionprocessing in accordance with the resolution of the viewfinder orperforms frame rate conversion processing. Furthermore, in a case wherean input signal of the viewfinder is an RGB signal, the viewfinderoutput unit 27 may be configured to directly acquire a γ converted imagesignal, which has not been subjected to color space conversionprocessing by the #1 camera signal processing unit 25-1.

The video output unit 26 acquires signal-processed video signals fromthe camera signal processing units 25 or decoded reproduction image datafrom the codec units 28, and performs frame combination, as necessary,to generate a video signal of a predetermined frame rate. The videooutput unit 26 outputs the video signal of the predetermined frame rateas a video output indicated by k in the figures to, for example, anexternal recording device or display device, a predetermined signaltransmission channel, or the like.

The details of frame combination for video image data to be output fromthe imaging apparatus 1 will be described below with reference to FIGS.6 and 7.

Next, the relationship between a captured image in the imaging apparatus1 and image frames to be processed and recorded will be explained withreference to FIG. 3.

Here, explanation will be given assuming that the imaging element 22 ofthe imaging apparatus 1 captures an image at HD resolution and 240 frameper second. Therefore, in FIG. 1, the imaging element output dataindicated by a in the figures is such that 240 frames of image data ofHD resolution are output per second in the time direction. FIG. 2illustrates frame distribution in a case where 12 frames including theN-th frame to the (N+11)-th frame are output as the imaging elementoutput data a from the imaging element 22.

As described above, the imaging element output data a is temporarilystored in the frame memory 24 by the memory control unit 23. At thetiming of time t1 when the N-th frame is output from the imaging element22, four frames of image data including the (N−4)-th to (N−1)-th framesare held in the frame memory 24 having a recording capacity of at leasteight frames. For a period until time t2 when four frames of image dataincluding the N-th frame, the (N+1)-th frame, the (N+2)-th frame, andthe (N+3)-th frame, which are output from the imaging element 22, aresequentially supplied to and recorded on the frame memory 24,specifically, for a period of 4/240 seconds, the memory control unit 23reads the four frames of image data including the (N−4)-th to (N−1)-thframes in parallel, which are stored in the frame memory 24, anddistributes the (N−4)-th frame to the #1 camera signal processing unit25-1, the (N−3)-th frame to the #2 camera signal processing unit 25-2,the (N−2)-th frame to the #3 camera signal processing unit 25-3, and the(N−1)-th frame to the #4 camera signal processing unit 25-4.

Then, at the timing of time t2 when the (N+4)-th frame is output fromthe imaging element 22, four frames of image data including the N-th to(N+3)-th frames are held in the frame memory 24. For a period until timet3 when four frames of image data including the (N+4)-th frame, the(N+5)-th frame, the (N+6)-th frame, and the (N+7)-th frame, which areoutput from the imaging element 22, are sequentially supplied to andrecorded on the frame memory 24, specifically, for a period of 4/240seconds, the memory control unit 23 reads the four frames of image dataincluding the N-th to (N+3)-th frames in parallel, which are stored inthe frame memory 24, and distributes the N-th frame to the #1 camerasignal processing unit 25-1, the (N+1)-th frame to the #2 camera signalprocessing unit 25-2, the (N+2)-th frame to the #3 camera signalprocessing unit 25-3, and the (N+3)-th frame to the #4 camera signalprocessing unit 25-4.

Subsequently, at the timing of time t3 when the (N+8)-th frame is outputfrom the imaging element 22, four frames of image data including the(N+4)-th to (N+7)-th frames are held in the frame memory 24. For aperiod until four frames of image data including the (N+8)-th frame, the(N+9)-th frame, the (N+10)-th frame, and the (N+11)-th frame, which areoutput from the imaging element 22, are sequentially supplied to andrecorded on the frame memory 24, specifically, for a period of 4/240seconds, the memory control unit 23 reads the four frames of image dataincluding the (N+4)-th to (N+7)-th frames in parallel, which are storedin the frame memory 24, and distributes the (N+4)-th frame to the #1camera signal processing unit 25-1, the (N+5)-th frame to the #2 camerasignal processing unit 25-2, the (N+6)-th frame to the #3 camera signalprocessing unit 25-3, and the (N+7)-th frame to the #4 camera signalprocessing unit 25-4.

That is, the imaging element output data a supplied from the imagingelement 22 is read from the frame memory 24 with a delay of four frames,and is supplied to one of the camera signal processing units 25. Each ofthe video image data b to e read from the frame memory 24 has a framerate that is ¼ that of the imaging element output data a. For example,when the imaging element output data a is image data of HD resolutionand 240 frames per second, each of the video image data b to e is imagedata of HD resolution and 60 frames per second.

Next, the input/output timings of captured image frames in the imagingapparatus 1 and image frames to be processed and recorded will beexplained with reference to FIG. 4.

The imaging element output data a illustrated in FIG. 1 includes, inaddition to pixel data that constitutes each captured image frame, incorrespondence with each frame, a vertical synchronization signal A thatis a synchronization signal representing the top of each frame, ahorizontal synchronization signal generated for each of horizontal linesconstituting a frame, and an enable signal. Here, when the imagingelement output data a has HD resolution and 240 frames per second, thevertical synchronization signal A is a signal that becomes active every1/240 seconds. The pixel data is a signal in which all pixel data items(for example, 2200 pixels by 1125 lines of pixels) are arranged in timeseries within one frame period ( 1/240 seconds) including a blankingperiod.

Then, the imaging element output data a is sequentially supplied to andstored in the frame memory 24. In parallel with this storage process,four frames of video image data, which have already been stored in theframe memory 24, are read in parallel, that is, alternately inpredetermined read units. Each of the video image data b to e read inparallel is, as described above, image data of HD resolution and 60frames per second. The read unit is determined on the basis of, forexample, the amount of data that can be exchanged each time the framememory 24 is accessed, and can be, for example, data of a predeterminednumber of bits, or one line or a plurality of predetermined lines ofdata within one frame of image.

A vertical synchronization signal B-1 that is a synchronization signalindicating the top of the video image data b supplied to the #1 camerasignal processing unit 25-1 is a signal that becomes active every 1/60seconds. The pixel data that constitutes the video image data b is asignal in which all pixel data items (for example, 2200 pixels by 1125lines of pixels) are arranged in time series within one frame period (1/60 seconds) including a blanking period.

Similarly to above, a vertical synchronization signal B-2 that is asynchronization signal indicating the top of the video image data csupplied to the #2 camera signal processing unit 25-2 is a signal thatbecomes active every 1/60 seconds. The pixel data that constitutes thevideo image data c is a signal in which all pixel data items (forexample, 2200 pixels by 1125 lines of pixels) are arranged in timeseries within one frame period ( 1/60 seconds) including a blankingperiod. Then, a vertical synchronization signal B-3 that is asynchronization signal indicating the top of the video image data dsupplied to the #3 camera signal processing unit 25-3 is a signal thatbecomes active every 1/60 seconds. The pixel data that constitutes thevideo image data d is a signal in which all pixel data items (forexample, 2200 pixels by 1125 lines of pixels) are arranged in timeseries within one frame period ( 1/60 seconds) including a blankingperiod. Additionally, a vertical synchronization signal B-4 that is asynchronization signal indicating the top of the video image data esupplied to the #4 camera signal processing unit 25-4 is a signal thatbecomes active every 1/60 seconds. The pixel data that constitutes thevideo image data e is a signal in which all pixel data items (forexample, 2200 pixels by 1125 lines of pixels) are arranged in timeseries within one frame period ( 1/60 seconds) including a blankingperiod.

Each of the video image data b to e read from the frame memory 24 by theprocess of the memory control unit 23 can be independently processed ordisplayed as video image data of HD resolution and 60 frames per second,and is supplied to and processed in the corresponding one of the #1camera signal processing unit 25-1, the #2 camera signal processing unit25-2, the #3 camera signal processing unit 25-3, and the #4 camerasignal processing unit 25-4.

FIG. 5 is a block diagram illustrating a detailed configuration of thecamera signal processing unit 25.

A WB (white balance) correction unit 51 corrects the balance ofrespective color components of RAW data, and performs white balanceadjustment so as to make brightness values of the red component (R), thegreen component (G), and the blue component (B) equal to each other inan achromatic area.

An RGB interpolation synchronization processing unit 52 performs aninterpolation process based on neighboring pixel data, that is,so-called demosaic processing, on RAW data having only one of the R, G,and B color components in each pixel, and outputs resulting data aspixel data having all the color components in each pixel. For example,in imaging data having the Bayer pattern illustrated in FIG. 2, a pixelB₂₂ at the pixel position (x=2, y=2) has only the B component. Thus, aninterpolation process using neighboring pixels is performed to generatethe R component and the G component which are output as R₂₂ and G₂₂,respectively. Similarity applies to a pixel in which only the Rcomponent exists as imaging data (for example, x=1, y=1) and a pixel inwhich only the G component exists (for example, x=2, y=1).

A matrix processing unit 53 is a color correction processing circuitusing a 3 by 3 matrix. The process thereof allows the color balance thatdepends upon the color space of the imaging element and the photographicenvironment to approach a true color space compatible with the signalstandard.

A gamma (γ) correction unit 54 performs gamma correction processingaccording to a video signal standard by using lookup table processingcorresponding to the number of input/output gradation levels.

A color space conversion unit 55 converts pixel data based on an RGBcolor space into that based on a YCbCr color space. Here, the colorspace after the conversion is defined by a standardization standard ofvideo signals. For example, the HDTV standard is specified inITU-R.BT709. Furthermore, the color space conversion unit 55 performsdegeneration processing on color difference signals Cb and Cr, performsconversion into the 4:2:2 format, and then outputs a converted videosignal (YCbCr 4:2:2).

In this way, a video signal (YCbCr 4:2:2) of HD resolution and 60 framesper second is output from each of the camera signal processing units 25.

In a case where the video image data processed by the camera signalprocessing units 25 are compression encoded and are thereafter recordedon the recording units 29, the video image data processed by the #1camera signal processing unit 25-1, the #2 camera signal processing unit25-2, the #3 camera signal processing unit 25-3, and the #4 camerasignal processing unit 25-4 are supplied to the corresponding ones ofthe #1 codec unit 28-1, the #2 codec unit 28-2, the #3 codec unit 28-3,and the #4 codec unit 28-4, and are compression encoded. The compressionencoded image data streams are respectively supplied to and recorded onthe #1 recording unit 29-1, the #2 recording unit 29-2, the #3 recordingunit 29-3, and the #4 recording unit 29-4.

Then, in a case where compression encoded image data streams recorded onthe recording units 29 are reproduced and output, the camera controlunit 11 reads compression encoded data from a predetermined recordingunit 29 on the basis of the frame rate of the video image data outputfrom the video output unit 26, and causes the compression encoded datato be decoded using the codec unit 28 and supplied to the video outputunit 26. The video output unit 26 performs frame combination, asnecessary, on the decoded video image data supplied thereto, andsupplies generated video image data to outside as a video output k.

Additionally, also in a case where captured video is directly output,the camera control unit 11 supplies, based on the frame rate of thevideo image data output from the video output unit 26, processed videoimage data from a predetermined camera signal processing unit 25 to thevideo output unit 26. The video output unit 26 performs framecombination on the supplied video image data, and supplies generatedvideo image data to outside as a video output k.

The output frame rate and frame combination will be explained usingFIGS. 6 and 7 using, as an example, a case where compression encodedimage data streams recorded on the recording units 29 are reproduced.

Each of the recording units 29 has recorded thereon an image data streamthat is generated by performing compression encoding after a movingimage of a high resolution (here, HD resolution) and a high frame rate(here, 240 frames per second) is divided into four segments in units offrames. In other words, the image data stream recorded on each of therecording units 29 is each a moving image of HD resolution and 60 framesper second, and is shifted in time by one frame with respect to acaptured image at 240 frames per second. Specifically, the image datastream recorded on the #1 recording unit 29-1 is a moving image of 60frames per second in which one out of four frames is extracted within amoving image of 240 frames per second, and the image data streamrecorded on the #2 recording unit 29-2 is image data of 60 frames persecond that is delayed in time by one frame with respect to the imagedata stream recorded on the #1 recording unit 29-1. Similarly, the imagedata stream recorded on the #3 recording unit 29-3 is image data of 60frames per second that is delayed in time by two frames with respect tothe image data stream recorded on the #1 recording unit 29-1, and theimage data stream recorded on the #4 recording unit 29-4 is image dataof 60 frames per second that is delayed in time by three frames withrespect to the image data stream recorded on the #1 recording unit 29-1.

In a case where the compression encoded data recorded on the recordingunits 29 is reproduced and output, under control of the camera controlunit 11, the recorded image data streams are sequentially read, startingfrom a specified frame, and are decoded and reproduced by thecorresponding ones of the codec units 28.

FIG. 6 illustrates frame combination in a case where video image data of240 frames per second is output.

In a case where a reproduction operation for an output of 240 frames persecond is commanded, the camera control unit 11 controls each of therecording units 29 so that the compression encoded image data streamrecorded thereon is read, starting from a specified beginning frame. Thefour reproduced image data streams are respectively decoded by the codecunits 28, and the four video image data items f to i, which have HDresolution and 60 frames per second and which are each shifted by 1/240seconds, that is, by one frame in video image data of 240 frames persecond, are input to the video output unit 26.

The video output unit 26 temporarily stores the four video image dataitems f to i in a frame memory (not illustrated), arranges them so as tohave HD resolution and a frame rate of 240 frames per second, and readsthem in the order of frames illustrated in FIG. 6, that is, in the sameorder of frames as that during the imaging of 240 frames per second tooutput them as video signals. In FIG. 6, N, N+1, N+2 . . . , whichindicates the order of frames, indicates the order of frames in the caseof video image data of 240 frames per second. That is, N, N+1, N+2 . . ., which indicates the order of frames in FIG. 6, indicates the order ofarrangement of frames that are shifted by only 1/240 seconds, which isbasically similar to the order of frames in the case of captured videoimage data of 240 frames per second, which has been explained using FIG.3.

Specifically, the N-th frame of the video image data item f is outputfrom the #1 codec unit 28-1, the (N+1)-th frame of the video image dataitem g is output from the #2 codec unit 28-2, the (N+2)-th frame of thevideo image data item h is output from the #3 codec unit 28-3, and the(N+3)-th frame of the video image data item i is output from the #4codec unit 28-4. The output frames are supplied to the video output unit26. Then, the video output unit 26 holds the four frames including theN-th to (N+3)-th frames in the frame memory (not illustrated).

Then, next, the (N+4)-th frame of the video image data item f is outputfrom the #1 codec unit 28-1, the (N+5)-th frame of the video image dataitem g is output from the #2 codec unit 28-2, the (N+6)-th frame of thevideo image data item h is output from the #3 codec unit 28-3, and the(N+7)-th frame of the video image data item i is output from the #4codec unit 28-4. The output frames are supplied to the video output unit26. The video output unit 26 holds the four frames including the(N+4)-th to (N+7)-th frames in the frame memory (not illustrated). Thevideo output unit 26 also arranges the four frames including the N-th to(N+3)-th frames, which have already been held in the frame memory, inthe order of frames, and outputs the four frames as a video output k.That is, the video output k is output with a delay of at least fourframes with respect to the timing when the supply of the video imagedata items g to i to the video output unit 26 is started.

In this way, the video signal k output from the video output unit 26allows image display at HD resolution and 240 frames per second. Bydoing so, the high-speed display of a moving image captured by theimaging apparatus 1 can be realized, as compared to a normal frame ratesuch as, for example, 60 frames per second, without performing complexprocessing on the moving image.

Here, explanation has been given assuming that the video output unit 26is designed to output a video signal of HD resolution and 240 frames persecond in one broadband video transmission channel. However, the videooutput unit 26 may be designed to output four-channel video signals ofHD resolution and 60 frames per second, which have not been subjected toframe combination, to a display unit (not illustrated) via four videotransmission channels in accordance with the specification of a displaydevice (not illustrated) that obtains a video signal k to be output. Inthe case of doing so, it is necessary for the display device thatobtains four-channel video signals of HD resolution and 60 frames persecond to execute frame combination for generating a video signal of HDresolution and 240 frames per second from the input four-channel videosignals of HD resolution and 60 frames per second in a manner similar tothat in the case explained using FIG. 6. Additionally, in this case, thevideo output unit 26 may output the video image data items f to i of HDresolution and 60 frames per second directly as four-channel videosignals without performing frame combination.

Next, the reproduction operation at 120 frames per second will beexplained with reference to FIG. 7.

For example, in a case where a video signal of HD resolution and 120frames per second is output due to the reason of a display device thatsupports only up to 120 frames per second, a video signal transmissionchannel that supports only up to 120 frames per second, or the like, thecamera control unit 11 causes two recording units 29 among the recordingunits 29, on which compression encoded image data streams that are eachshifted by two frames in movie frames of 240 frames per second arerecorded, to supply the compression encoded image data stream recordedthereon to the codec units 28. The two reproduced image data streams arerespectively decoded by the codec units 28. Then, the two video imagedata items f and h (or the video image data items g and i), which haveHD resolution and 60 frames per second and each of which is shifted by2/240 seconds, that is, by two frames in video image data of 240 framesper second, are input to the video output unit 26.

The video output unit 26 temporarily stores the two video image dataitems f and h in the frame memory (not illustrated), arranges them so asto have HD resolution and a frame rate of 120 frames per second, readsthem in the order of frames illustrated in FIG. 7, and outputs them asvideo signals.

Also in FIG. 7, N, N+2, N+4 . . . , which indicates the order of frames,indicates the order of frames in the case of video image data of 240frames per second. That is, since the order of frames of the video imagedata items subjected to frame combination is N, N+2, N+4 . . . , videoimage data obtained after the frame combination becomes equal to videoimage data of 120 frames per second that is obtained by decimating thevideo image data of 240 frames per second by ½.

In this way, the video output unit 26 can output not only a video outputk that allows image display at HD resolution and 240 frames per secondbut also a video output k that allows image display at HD resolution and120 frames per second without performing complex processing. By doingso, the imaging apparatus 1 can easily generate and output video imagedata of a plurality of frame rates.

Here, explanation has been given assuming that the video output unit 26is designed to output a video signal of HD resolution and 120 frames persecond in one broadband video transmission channel in accordance withthe specification of the display device (not illustrated) that acquiresa video signal k to be output. However, for example, the video outputunit 26 may be designed to output two-channel video signals of HDresolution and 60 frames per second to a display unit (not illustrated)via two video transmission channels. In the case of doing so, it isnecessary for the display device that acquires two-channel video signalsof HD resolution and 60 frames per second to execute frame combinationfor generating a video signal of HD resolution and 120 frames per secondfrom the input two-channel video signals of HD resolution and 60 framesper second in a manner similar to that in the case explained using FIG.7. Additionally, in this case, the video output unit 26 may output thevideo image data items f and h of HD resolution and 60 frames per seconddirectly as two-channel video signals without performing framecombination.

Also not illustrated in the figures, furthermore, in a case where thedisplay device supports only a normal frame rate, namely, 60 frames persecond, in a case where the transmission channel supports only 60 framesper second, or any other case, the camera control unit 11 controls therespective units to read a compression encoded image data stream fromone of the recording units 29, to decode the compression encoded imagedata stream using the corresponding one of the codec units 28, and tosupply video image data of 60 frames per second to the video output unit26. The video output unit 26 outputs the supplied video image data of HDresolution and 60 frames per second.

In this way, the imaging apparatus 1 performs imaging at a highresolution and a high frame rate, divides a captured moving image inunits of frames, and performs parallel processing on resulting datasegments. Thereby, image processing, codec, or recording processing formoving image data can be executed at a normal frame rate (for example,60 frames per second while imaging is performed at 240 frames persecond). During reproduction, video data can be output at a plurality ofoutput frame rates without performing complex processing.

Additionally, in a case where a captured image is output to anddisplayed on the viewfinder (not illustrated), the image data processedby the #1 camera signal processing unit 25-1 is output to the viewfinderoutput unit 27. Furthermore, in a case where image data to be output asvideo is output to and displayed on the viewfinder (not illustrated),the image data decoded by the #1 codec unit 28-1 is output to theviewfinder output unit 27. In other words, image data output to theviewfinder has a frame rate of 60 frames per second.

In contrast, for example, in a case where the viewfinder has thecapabilities to display a moving image of a frame rate of 120 frames persecond, the image data processed by the #3 camera signal processing unit25-3 in addition to the image data processed by the #1 camera signalprocessing unit 25-1 may be output to the viewfinder output unit 27, orthe image data decoded by the #3 codec unit 28-3 in addition to theimage data decoded by the #1 codec unit 28-1 may be output to theviewfinder output unit 27 to perform frame combination and output aresult.

Additionally, it goes without saying that in a case where the viewfinderhas the capabilities to display a moving image of a frame rate of 240frames per second, under control of the camera control unit 11, fourprocessed video image data items of 60 frames per second may be suppliedfrom all the camera signal processing units 25 to the video output unit26, or compression encoded image data streams may be read from all therecording units 29 and may be respectively decoded using the codec units28. Four video image data items of 60 frames per second may be suppliedto the video output unit 26 and subjected to frame combination, and aresult may be displayed.

Additionally, in a case where captured and processed image data isoutput to the outside without being temporarily compressed and recorded,based on the frame rate of the video image data to be output, the videoimage data processed by a predetermined camera signal processing unit 25among the #1 camera signal processing unit 25-1, the #2 camera signalprocessing unit 25-2, the #3 camera signal processing unit 25-3, and the#4 camera signal processing unit 25-4 is supplied to the video outputunit 26.

Specifically, in a case where the video image data to be output has aframe rate of 240 frame per second, video image data is supplied to thevideo output unit 26 from all the #1 camera signal processing unit 25-1,the #2 camera signal processing unit 25-2, the #3 camera signalprocessing unit 25-3, and the #4 camera signal processing unit 25-4.Then, in a case where the video image data to be output has a frame rateof 120 frames per second, video image data is supplied to the videooutput unit 26 from two camera signal processing units 25 among the #1camera signal processing unit 25-1, the #2 camera signal processing unit25-2, the #3 camera signal processing unit 25-3, and the #4 camerasignal processing unit 25-4, which process frames that are each shiftedby two frames in movie frames of 240 frames per second. Additionally, ina case where the video image data to be output has a frame rate of 60frames per second, video image data is supplied to the video output unit26 from one of the #1 camera signal processing unit 25-1, the #2 camerasignal processing unit 25-2, the #3 camera signal processing unit 25-3,and the #4 camera signal processing unit 25-4.

The imaging apparatus 1 described above is configured to capture amoving image at 240 frames per second that is four times 60 frames persecond, which is generally widely used for capturing a moving image, todivide captured moving image data into four pieces for each frame togenerate four-channel moving image data of 60 frames per second, and toperform various processes so that a moving image of 60 frames persecond, 120 frames per second, or 240 frames per second can bereproduced. However, it goes without saying that any other imaging framerate or any other number of segments of captured moving image data maybe used.

Specifically, for example, a moving image captured at 240 frames persecond may be divided into two or three segments, or a moving imagecaptured at 120 frames per second may be divided into two or threesegments. Additionally, a moving image captured at 200 frames per secondmay be divided into four segments, or a moving image captured at 100frames per second may be divided into two segments. Alternatively, amoving image captured at 96 frames per second may be divided into foursegments, or a moving image captured at 48 frames per second may bedivided into two segments.

At this time, when moving image data segments of each channel have aframe rate that is generally widely used for capturing a moving image,such as, for example, 60 frames per second, 50 frames per second, or 24frames per second, for example, a general-purpose product can be usedfor a circuit or the like necessary for signal processing or codec, andcost can be reduced, which is preferable. However, the frame rate ofmoving image data segments of each channel may be any other value.

Additionally, here, explanation has been given of a case where, by wayof example, the frame rates of segments of each channel are equal toeach other. However, it goes without saying that segments of eachchannel may have different frame rates. For example, a moving imagecaptured at 240 frames per second may be divided into one channel for120 frames per second and two channels for 60 frames per second.

Note that it goes without saying that in a case where the recordingunits 29 have a large capacity and a high recording rate, in a casewhere image degradation due to codec is desired to be avoided, or anyother case, the codec units 28 may be omitted so that video image datathat is not compressed or encoded may be recorded on the recording units29.

In this manner, the imaging apparatus 1 divides image data captured at aframe rate that is N times (here, four times) a frame rate generallywidely used for capturing a moving image, such as, for example, 60frames per second, 50 frames per second, or 24 frames per second, into Nsegments in the time direction in units of frames, and is therebycapable of processing or recording high-frame-rate video image data as Nnormal-frame-rate image data items. Furthermore, the imaging apparatus 1is capable of outputting moving image data at a plurality of frame rateswithout performing complex processing.

Additionally, the imaging apparatus 1 includes, as the imaging element22, a single-plate color solid-state imaging element having, on a lightreceiving surface, a color filter that transmits light in differentwaveform ranges for individual pixels. However, it goes without sayingthat any other scheme may be used as the imaging method. For example,the present invention can also be applied in a three-plate, instead ofsingle-plate color, imaging apparatus.

FIG. 9 is a block diagram illustrating a configuration of a three-plateimaging apparatus 101 that uses three solid-state imaging elements.

Note that portions corresponding to those of the imaging apparatus 1 ofFIG. 1 are assigned the same numerals, and explanation thereof isomitted as appropriate. That is, the imaging apparatus 101 has aconfiguration basically similar to that of the imaging apparatus 1explained using FIG. 1, except that a camera control unit 111 isprovided in place of the camera control unit 11, imaging elements 121 to123 are provided in place of the imaging element 22, a memory controlunit 124 is provided in place of the memory control unit 23, a framememory 125 is provided in place of the frame memory 24, a #1 camerasignal processing unit 126-1, a #2 camera signal processing unit 126-2,a #3 camera signal processing unit 126-3, and a #4 camera signalprocessing unit 126-4 are provided in place of the #1 camera signalprocessing unit 25-1, the #2 camera signal processing unit 25-2, the #3camera signal processing unit 25-3, and the #4 camera signal processingunit 25-4, and a dichroic prism 120 that splits light incident throughthe imaging optical system 21 is newly provided.

In the following explanation, each of the #1 camera signal processingunit 126-1, the #2 camera signal processing unit 126-2, the #3 camerasignal processing unit 126-3, and the #4 camera signal processing unit126-4 is referred to simply as a camera signal processing unit 126unless they need to be individually identified.

The camera control unit 111 controls the operation of the respectiveunits of the imaging apparatus 101.

The imaging elements 121, 122, and 123 receive light that is incidentthrough the imaging optical system 21 and that is split by the dichroicprism 120 into the red color component (R), the green color component(G), and the blue color component (B). Here, it is assumed that theimaging element 121 receives light in a wavelength range centered on thered color component (R), the imaging element 122 receives light in awavelength range centered on the green color component (G), and theimaging element 123 receives light in a wavelength range centered on theblue color component (B).

Under control of the camera control unit 111, the memory control unit124 supplies imaging data corresponding to the wavelength rangescentered on the respective RGB color components, which are supplied fromthe imaging elements 121, 122, and 123, to the frame memory 125 in sucha manner that one frame contains three colors, and also divides fourframes of imaging data, each frame containing three colors, which arerecorded on the frame memory 125, in units of frames using processingsimilar to that in the case explained using FIGS. 3 and 4 torespectively supply resulting segments to the camera signal processingunits 126. The frame memory 125 has a storage capacity capable ofholding at least eight frames of imaging data, each frame containing RGBthree colors.

Therefore, in the three-plate imaging apparatus 101 that performsimaging using imaging elements, since imaging data of each of R, G, andB can be obtained at a pixel position, it is not necessary for each ofthe camera signal processing units 126 to perform RGB interpolationsynchronization processing.

A configuration of the camera signal processing units 126 is illustratedin FIG. 9.

Note that portions corresponding to those of the camera signalprocessing unit 25 of FIG. 5 are assigned the same numerals, andexplanation thereof is omitted as appropriate. That is, the camerasignal processing unit 126 has a configuration basically similar to thatof the camera signal processing unit 25 of FIG. 5, except that the RGBinterpolation synchronization processing unit 52 is omitted and a WBcorrection unit 151 is provided in place of the WB correction unit 51.

The WB correction unit 151 is supplied with an image signal having R, G,and B for each pixel, performs R-G-B color balance adjustment for eachpixel, and supplies adjusted image signals (R, G, B) to a matrixprocessing unit 53. The signal processing subsequent to the matrixprocessing is basically similar to that of the camera signal processingunit 25 explained using FIG. 5.

In this manner, the imaging apparatus 101 is also configured to capturea moving image at 240 frames per second that is four times 60 frames persecond, which is generally widely used for capturing a moving image, todivide captured moving image data into four pieces for each frame togenerate four-channel moving image data of 60 frames per second, and toperform various processes so that a moving image of 60 frames persecond, 120 frames per second, or 240 frames per second can bereproduced. However, it goes without saying that any other imaging framerate or any other number of segments of captured moving image data maybe used.

Specifically, for example, a moving image captured at 240 frames persecond may be divided into two or three segments, or a moving imagecaptured at 120 frames per second may be divided into two or threesegments. Additionally, a moving image captured at 200 frames per secondmay be divided into four segments, or a moving image captured at 100frames per second may be divided into two segments. Alternatively, amoving image captured at 96 frames per second may be divided into foursegments, or a moving image captured at 48 frames per second may bedivided into two segments.

At this time, when moving image data segments of each channel have aframe rate that is generally widely used for capturing a moving image,such as, for example, 60 frames per second, 50 frames per second, or 24frames per second, for example, a general-purpose product can be usedfor a circuit or the like necessary for signal processing or codec, andcost can be reduced, which is preferable. However, the frame rate ofmoving image data segments of each channel may be any other value.

Additionally, also here, explanation has been given of a case where, byway of example, the frame rates of segments of each channel are equal toeach other. However, it goes without saying that segments of eachchannel may have different frame rates. For example, a moving imagecaptured at 240 frames per second may be divided into one channel for120 frames per second and two channels for 60 frames per second.

Note that, also in the imaging apparatus 101, it goes without sayingthat in a case where the recording units 29 have a large capacity and ahigh recording rate, in a case where image degradation due to codec isdesired to be avoided, or any other case, the codec units 28 may beomitted so that video image data that is not compressed or encoded maybe recorded on the recording units 29.

In this manner, like the imaging apparatus 1, the imaging apparatus 101also divides image data captured at a frame rate that is N times (here,four times) a frame rate generally widely used for capturing a movingimage, such as, for example, 60 frames per second, 50 frames per second,or 24 frames per second, into N segments in the time direction in unitsof frames, and is thereby capable of processing or recording ahigh-frame-rate video signal as N normal-frame-rate image data items.Furthermore, the imaging apparatus 1 is capable of outputting movingimage data at a plurality of frame rates without performing complexprocessing.

Next, an imaging/recording/output process executed by the imagingapparatus 1 or the imaging apparatus 101 will be explained withreference to a flowchart of FIG. 10.

Note that although, in the flowchart of FIG. 10, for ease ofunderstanding, each of the imaging, recording, and outputting processesis explained in terms of individual steps, it goes without saying thatthese processes can be executed in parallel in the imaging apparatus 1or the imaging apparatus 101.

In step S1, the camera control unit 11 or the camera control unit 111determines whether or not starting of capturing a moving image has beencommanded from an operation input unit (not illustrated) or the like. Ina case where it is determined in step S1 that starting of capturing amoving image has not been commanded, the process proceeds to step S13described below.

In a case where it is determined in step S1 that starting of capturing amoving image has been commanded, in step S2, the camera control unit 11or the camera control unit 111 controls the respective units to executean imaging process.

Specifically, the camera control unit 11 controls the imaging opticalsystem 21 to cause light, which corresponds to an image to be captured,to be incident on the imaging element 22. Under control of the cameracontrol unit 11, the imaging element 22 acquires an image signalconfigured by a pixel data having a color pattern according to a Bayerpattern as illustrated in FIG. 2 at a high resolution (here, forexample, HD resolution) and a high frame rate (here, for example, 240frames per second, that is, referred to as 240 Hz), and supplies theimage signal to the memory control unit 23. Additionally, the cameracontrol unit 111 controls the imaging optical system 21 to split thelight corresponding to an image to be captured into the red colorcomponent (R), the green color component (G), and the blue colorcomponent (B) using the dichroic prism 120, and thereafter causes thecomponents to be incident on the imaging elements 121 to 123,respectively. Under control of the camera control unit 111, the imagingelements 121 to 123 acquire image signals by receiving lightcorresponding to the wavelength ranges of the respective RGB colorcomponents at a high resolution (here, for example, HD resolution) and ahigh frame rate (here, for example, 240 frames per second, that is,referred to as 240 Hz), and supply the image signals to the memorycontrol unit 23.

In step S3, the camera control unit 11 or the camera control unit 111controls the respective units to start or continue the process ofdividing imaging data.

Specifically, under control of the camera control unit 11 or the cameracontrol unit 111, the memory control unit 23 or the memory control unit124 divides the imaging data supplied from the imaging element 22 or theimaging elements 121, 122, and 123 in units of frames in the mannerexplained using FIGS. 3 and 4, and respectively supplies resultingimaging data segments to the camera signal processing units 25 or thecamera signal processing units 126.

In step S4, the camera signal processing units 25 or the camera signalprocessing units 126 execute signal processing under control of thecamera control unit 11 or the camera control unit 111.

Specifically, as explained using FIG. 5, the camera signal processingunits 25 correct the white balance of RAW data supplied thereto, executeRGB interpolation synchronization, that is, demosaic processing, andperform matrix processing and γ correction. Thereafter, the camerasignal processing units 25 perform color space conversion, and outputgenerated video signals (YCbCr 4:2:2) of HD resolution and 60 frames persecond. Additionally, as explained using FIG. 9, the camera signalprocessing units 126 correct the white balance of image data constitutedby the respective RGB components, and perform matrix processing and γcorrection. Thereafter, the camera signal processing units 126 performcolor space conversion, and output generated video signals (YCbCr 4:2:2)of HD resolution and 60 frames per second.

In step S5, the camera control unit 11 or the camera control unit 111determines whether or not the output of an image that is being capturedto the viewfinder has been commanded.

In a case where it is determined in step S5 that the output to theviewfinder has been commanded, in step S6, the camera control unit 11 orthe camera control unit 111 controls the process of outputtingsignal-processed video data to the viewfinder.

Specifically, under control of the camera control unit 11 or the cameracontrol unit 111, the viewfinder output unit 27 acquires the videosignal output from the #1 camera signal processing unit 25-1 or the #1camera signal processing unit 126-1, and converts the video signal intoa signal that can be displayed on the viewfinder. Thereafter, theviewfinder output unit 27 outputs the signal to the viewfinder, which isnot illustrated, to display a moving image of a frame rate of 60 framesper second.

Note that an input signal of the viewfinder is an RGB signal, theviewfinder output unit 27 may be configured to directly acquire an RGBsignal, which has not been subjected to color space conversionprocessing by the #1 camera signal processing unit 25-1 or the #1 camerasignal processing unit 126-1. Additionally, when the frame rate of amoving image that can be displayed on the viewfinder is a frame rate of,for example, 120 frames per second, the viewfinder output unit 27 may bedesigned to acquire the video signals output from the #1 camera signalprocessing unit 25-1 and the #3 camera signal processing unit 25-3 orthe #1 camera signal processing unit 126-1 and the #3 camera signalprocessing unit 126-3 (or the #2 camera signal processing unit 25-2 andthe #4 camera signal processing unit 25-4 or the #2 camera signalprocessing unit 126-2 and the 4 camera signal processing unit 126-4), toperform frame combination, and to output a result to the viewfinder (notillustrated). Alternatively, when the frame rate of a moving image thatcan be displayed on the viewfinder is a frame rate of, for example, 240frames per second, the viewfinder output unit 27 may be designed toacquire the video signals output from all the camera signal processingunits 25 or all the camera signal processing units 126, to perform framecombination, and to output a result to the viewfinder (not illustrated).

In a case where it is determined in step S5 that the output to theviewfinder has not been commanded, or after the completion of theprocessing of step S6, in step S7, the camera control unit 11 or thecamera control unit 111 determines whether or not the video output of animage that is being captured has been commanded.

In a case where it is determined in step S7 that the video output of animage that is being captured has been commanded, in step S8, the cameracontrol unit 11 or the camera control unit 111 starts a video outputprocess.

Specifically, under control of the camera control unit 11 or the cameracontrol unit 111, the video output unit 26 acquires signal-processedvideo signals from the camera signal processing units 25 or the camerasignal processing units 126, performs frame combination, as necessary,to generate a video signal of a predetermined frame rate, and outputsthe video signal as a video output indicated by k in the figures to, forexample, an external recording device or display device, a predeterminedsignal transmission line, or the like.

At this time, under control of the camera control unit 11 or the cameracontrol unit 111, the video output unit 26 acquires video data based onthe frame rate of a video signal to be output from the camera signalprocessing units 25 or the camera signal processing units 126, andperforms frame combination.

Specifically, in a case where moving image data of a frame rate of 60frames per second is output, processed video image data of 60 frames persecond is supplied from one of the camera signal processing units 25 orone of the camera signal processing units 126 to the video output unit26, and is supplied to the video output unit 26. The video output unit26 outputs the supplied video image data of 60 frames per second.

Additionally, in a case where moving image data of a frame rate of 120frames per second is output, processed two-channel video image data of60 frames per second is supplied to the video output unit 26 from twocamera signal processing units 25 among the camera signal processingunits 25 or the camera signal processing units 126, which process framesthat are each shifted by two frames in movie frames of 240 frames persecond. As explained using FIG. 7, the video output unit 26 performsframe combination so that the supplied two-channel video image data of60 frames per second can be alternately arranged on a frame-by-framebasis, and outputs combined video image data.

Additionally, in a case where moving image data of a frame rate of 240frames per second is output, processed four-channel video image data of60 frames per second is supplied to the video output unit 26 from allthe camera signal processing units 25 or all the camera signalprocessing units 126. As explained using FIG. 6, the video output unit26 performs frame combination so that the supplied four-channel videoimage data of 60 frames per second can be sequentially arranged on aframe-by-frame basis, and outputs combined video image data.

In a case where it is determined in step S7 that the video output hasnot been commanded, or after the completion of the processing of stepS8, in step S9, the camera control unit 11 or the camera control unit111 determines whether or not the recording of the imaging data has beencommanded.

In a case where it is determined in step S9 that the recording of theimaging data has been commanded, in step S10, under control of thecamera control unit 11 or the camera control unit 111, the codec units28 execute the process of encoding the signal-processed video image datasupplied from the camera signal processing units 25 or the camera signalprocessing units 126.

In step S11, the codec units 28 supply compression encoded image datastreams to the recording units 29 for recording.

In a case where it is determined in step S9 that the recording of theimaging data has not been commanded, or after the completion of theprocessing of step S11, in step S12, the camera control unit 11 or thecamera control unit 111 determines whether or not completion ofcapturing a moving image has been commanded. In a case where it isdetermined in step S12 that completion of capturing a moving image hasnot been commanded, the process returns to step S2, and the subsequentprocessing is repeated.

In a case where it is determined in step S1 that starting of capturing amoving image has not been commanded or in a case where it is determinedin step S12 that completion of capturing a moving image has beencommanded, in step S13, the camera control unit 11 or the camera controlunit 111 determines whether or not reproduction and output of the movingimage recorded on the recording units 29 have been commanded.

In a case where it is determined in step S13 that reproduction andoutput of the moving image recorded on the recording units 29 have beencommanded, in step S14, the camera control unit 11 or the camera controlunit 111 starts a video output process.

Specifically, as explained using FIGS. 6 and 7, under control of thecamera control unit 11 or the camera control unit 111, the video outputunit 26 acquires decoded reproduction image data from the codec units28, performs frame combination, as necessary, to generate a video signalof a predetermined frame rate, and outputs the video signal as a videooutput indicated by k in the figures to, for example, an externalrecording device or display device, a predetermined signal transmissionline, or the like.

At this time, under control of the camera control unit 11 or the cameracontrol unit 111, the video output unit 26 acquires video data based onthe frame rate of a video signal to be output from the codec units 28,and performs frame combination.

Specifically, in a case where moving image data of a frame rate of 60frames per second is output, a compression encoded image data stream isread from one of the recording units 29, and is decoded by thecorresponding one of the codec units 28 so that video image data of 60frames per second is supplied to the video output unit 26. The videooutput unit 26 outputs the supplied video image data of 60 frames persecond.

Additionally, in a case where moving image data of a frame rate of 120frames per second is output, a compression encoded image data stream isread from two recording units 29 among the recording units 29, on whichcompression encoded image data streams that are each shifted by twoframes in movie frames of 240 frames per second are recorded, and isdecoded by the corresponding one of the codec units 28 so that two videoimage data items of 60 frames per second are supplied to the videooutput unit 26. As explained using FIG. 7, the video output unit 26performs frame combination so that the two supplied video image dataitems of 60 frames per second can alternately be arranged on aframe-by-frame basis, and outputs combined video image data.

Additionally, in a case where moving image data of a frame rate of 240frames per second is output, compression encoded image data streams areread from all the recording units 29, and are respectively decoded bythe codec units 28 so that four video image data items of 60 frames persecond are supplied to the video output unit 26. As explained using FIG.6, the video output unit 26 performs frame combination so that the foursupplied video image data items of 60 frames per second can besequentially arranged on a frame-by-frame basis, and outputs combinedvideo image data.

Furthermore, when the viewfinder output of a moving image to bereproduced and output has been commanded, under control of the cameracontrol unit 11 or the camera control unit 111, the viewfinder outputunit 27 receives the video signal output from the #1 codec unit 28-1,converts the video signal into a signal that can be displayed on theviewfinder, and thereafter outputs the signal to the viewfinder (notillustrated) to display a moving image of a frame rate of 60 frames persecond.

In step S15, the camera control unit 11 or the camera control unit 111determines whether or not completion of reproducing and outputting therecorded moving image has been commanded. In a case where it isdetermined in step S15 that completion of reproducing and outputting therecorded moving image has not been commanded, the process returns tostep S14, and the subsequent processing is repeated.

In a case where it is determined in step S13 that reproduction andoutput of the recorded moving image have not been commanded or in a casewhere it is determined in step S15 that completion of reproducing andoutputting the recorded moving image has been commanded, the processends.

With such a process as above, in the imaging apparatus 1 or the imagingapparatus 101, a movie having a high frame rate is captured and captureddata is divided in units of frames. Thereby, moving image data having ahigh frame rate can be subjected to signal processing or encoding andcan be recorded without using complex processing or a high-speed signalprocessing circuit.

Moreover, even if the frame rate that matches the display capabilitiesof the viewfinder is lower than the frame rate of an image to becaptured, a movie captured at a frame rate that matches the displaycapabilities of the viewfinder can be displayed on the viewfinderwithout performing complex processing. Additionally, even in a casewhere the frame rate of desired output data is different from the framerate of recorded data, data of a predetermined frame rate can be outputwithout performing complex processing.

Next, an imaging data dividing process executed by the imaging apparatus1 or the imaging apparatus 101 will be explained with reference to aflowchart of FIGS. 11 and 12.

In step S41, the camera control unit 11 or the camera control unit 111sets a write start address of imaging data to be written from the memorycontrol unit 23 to the frame memory 24 or from the memory control unit124 to the frame memory 125.

In step S42, the camera control unit 11 or the camera control unit 111starts or continues block writing of the imaging data from the memorycontrol unit 23 to the frame memory 24 or from the memory control unit124 to the frame memory 125. Here, the term block is the unit of datathat can be exchanged once the frame memory 24 or the frame memory 125is accessed, and is here assumed to be one line of pixel data.

In step S43, the camera control unit 11 or the camera control unit 111determines whether or not four frames of imaging data have been writtenin the frame memory 24 or the frame memory 125. In a case where it isdetermined in step S43 that four frames of imaging data have not beenwritten, the process returns to step S42, and the subsequent processingis repeated.

In a case where it is determined in step S43 that four frames of imagingdata have been written, in step S44, the camera control unit 11 or thecamera control unit 111 sets, for four frames, read start addresses ofimaging data to be read from the frame memory 125.

Here, the read start addresses of the imaging data to be read from theframe memory 24 or the frame memory 125 coincide with the beginningaddresses of the respective four frames of imaging data written in theframe memory 24 or the frame memory 125. That is, the read startaddresses of the four frames of imaging data are set so that the fourframes of imaging data written in the frame memory 24 or the framememory 125 on the basis of the write start address set in the processingof step S41 or S46, which will be described below, can sequentially beread in parallel.

In step S45, the camera control unit 11 or the camera control unit 111initializes the value of a counter C used for determining a destinationto which the imaging data read from the frame memory 24 or the framememory 125 is supplied.

In step S46, the camera control unit 11 or the camera control unit 111sets a write start address of imaging data to be written next from thememory control unit 23 to the frame memory 24 or from the memory controlunit 124 to the frame memory 125. In step S46, a write start address forwriting new four frames of imaging data in an area different from arecording area of the four frames of imaging data written in the framememory 24 or the frame memory 125 presently.

In step S47, the camera control unit 11 or the camera control unit 111starts or continues block writing of the imaging data from the memorycontrol unit 23 to the frame memory 24 or from the memory control unit124 to the frame memory 125 on the basis of the value of the writeaddress set in step S46 or a value of a write address that isincremented in the processing described below.

In step S48, the camera control unit 11 or the camera control unit 111determines whether or not the value of the counter C used fordetermining a destination to which the imaging data read from the framememory 24 or the frame memory 125 is supplied meets the condition C=0.

In a case where it is determined in step S48 that the value of thecounter C meets the condition C=0, in step S49, the camera control unit11 or the camera control unit 111 causes the imaging data to be read inblocks from the frame memory 24 to the memory control unit 23 or fromthe frame memory 125 to the memory control unit 124 on the basis of theread address of the temporally earliest frame among the four frames ofimaging data that have been written in the frame memory 24 or the framememory 125, that is, whose read addresses have been set. Under controlof the camera control unit 11 or the camera control unit 111, the memorycontrol unit 23 or the memory control unit 124 supplies one block ofimaging data, which has been read therein, to the #1 camera signalprocessing unit 25-1 or the #1 camera signal processing unit 126-1, andthe process proceeds to step S55.

In a case where it is determined in step S48 that the value of thecounter C does not meet the condition C=0, in step S50, the cameracontrol unit 11 or the camera control unit 111 determines whether or notthe value of the counter C used for determining a destination to whichthe imaging data read from the frame memory 24 or the frame memory 125is supplied meets the condition C=1.

In a case where it is determined in step S50 that the value of thecounter C meets the condition C=1, in step S51, the camera control unit11 or the camera control unit 111 causes the imaging data to be read inblocks from the frame memory 24 to the memory control unit 23 or fromthe frame memory 125 to the memory control unit 124 on the basis of theread address of the temporally second earliest frame among the fourframes of imaging data that have been written in the frame memory 24 orthe frame memory 125, that is, whose read addresses have been set. Undercontrol of the camera control unit 11 or the camera control unit 111,the memory control unit 23 or the memory control unit 124 supplies oneblock of imaging data, which has been read therein, to the #2 camerasignal processing unit 25-2 or the #2 camera signal processing unit126-2, and the process proceeds to step S55.

In a case where it is determined in step S50 that the value of thecounter C does not meet the condition C=1, in step S52, the cameracontrol unit 11 or the camera control unit 111 determines whether or notthe value of the counter C used for determining a destination to whichthe imaging data read from the frame memory 24 or the frame memory 125is supplied meets the condition C=2.

In a case where it is determined in step S52 that the value of thecounter C meets the condition C=2, in step S53, the camera control unit11 or the camera control unit 111 causes the imaging data to be read inblocks from the frame memory 24 to the memory control unit 23 or fromthe frame memory 125 to the memory control unit 124 on the basis of theread address of the temporally third earliest frame among the fourframes of imaging data that have been written in the frame memory 24 orthe frame memory 125, that is, whose read addresses have been set. Undercontrol of the camera control unit 11 or the camera control unit 111,the memory control unit 23 or the memory control unit 124 supplies oneblock of imaging data, which has been read therein, to the #3 camerasignal processing unit 25-3 or the #3 camera signal processing unit126-3, and the process proceeds to step S55.

In a case where it is determined in step S52 that the value of thecounter C does not meet the condition C=2, the value of the counter Cmeets the condition C=3. Thus, in step S54, the camera control unit 11or the camera control unit 111 causes the imaging data to be read inblocks from the frame memory 24 to the memory control unit 23 or fromthe frame memory 125 to the memory control unit 124 on the basis of theread address of the temporally last frame among the four frames ofimaging data that have been written in the frame memory 24 or the framememory 125, that is, whose read addresses have been set. Under controlof the camera control unit 11 or the camera control unit 111, the memorycontrol unit 23 or the memory control unit 124 supplies one block ofimaging data, which has been read therein, to the #4 camera signalprocessing unit 25-4 or the #4 camera signal processing unit 126-4, andthe process proceeds to step S55.

After the completion of the processing of step S49, S51, S53, or S54, instep S55, the camera control unit 11 or the camera control unit 111increments the value of the counter C used for determining a destinationto which the imaging data read from the frame memory 24 or the framememory 125 is supplied. Here, when the value of the counter C is 3, thecounter C is initialized to 0.

In step S56, the camera control unit 11 or the camera control unit 111determines whether or not the reading of the four frames whose readaddresses have been set has been completed. In a case where it isdetermined in step S56 that the reading of the four frames whose readaddresses have been set has been completed, the process returns to stepS44, and the subsequent processing is repeated.

In a case where it is determined in step S56 that the reading of thefour frames whose read addresses have been set has not been completed,in step S57, the camera control unit 11 or the camera control unit 111increments the read address corresponding to the frame read in theprocessing of step S49, S51, S53, or S54.

In step S58, the camera control unit 11 or the camera control unit 111determines whether or not the writing of one frame, whose write startaddress has been set in step S46, in the frame memory 24 or the framememory 125 has been completed. In a case where it is determined in stepS58 that the writing of one frame has been completed, the processreturns to step S46, and the subsequent processing is repeated.

In a case where it is determined in step S58 that the writing of oneframe has not been completed, in step S59, the camera control unit 11 orthe camera control unit 111 increments the write address.

In step S60, the camera control unit 11 or the camera control unit 111determines whether or not completion of imaging has been commanded.

In a case where it is determined in step S60 that completion of imaginghas not been commanded, the process returns to step S47, and thesubsequent processing is repeated. In a case where it is determined instep S60 that completion of imaging has been commanded, the processends.

With such a process as above, captured video image data is divided intofour video image data segments of a frame rate that is one quarter theimaging frame rate, and the video image data segments are respectivelysupplied to the four camera signal processing units in parallel. Inother words, captured video image data is such that portions of the dataconstituting the respective frames are sequentially supplied to the fourcamera signal processing units in parallel so that a frame can besupplied to each of the four camera signal processing units. Therefore,the imaging apparatus 1 and the imaging apparatus 101 forms N parallelstreams of imaging data of a high resolution and a high frame rate thatis N times (for example, four times) a frame rate generally widely usedfor capturing a moving image (for example, 60 frames per second),thereby being capable of performing signal processing, image compressionprocessing, and recording processing similar to those in a case where animage is captured at a normal frame rate.

Additionally, the imaging apparatus 1 or the imaging apparatus 101 iscapable of outputting imaging data as video in the processing of step S8of FIG. 10 or outputting recorded data as video in the processing ofstep S14 of FIG. 10. As described above, the frame rate required forvideo output may be different from the imaging frame rate. Specifically,in a case where the imaging frame rate is 4M frames per second (where Mis, for example, a value corresponding to a frame rate generally widelyused for capturing a moving image), the frame rate of a video signal tobe output may be M frames per second, 2M frames per second, or 4M framesper second.

Next, a video output process executed in the imaging apparatus 1 or theimaging apparatus 101 will be explained with reference to a flowchart ofFIGS. 13 and 14.

In step S101, the camera control unit 11 or the camera control unit 111acquires a set value of an output frame rate. Here, explanation will begiven of a case where an image is captured at a frame rate of 240 framesper second, four data segments are subjected to frame combination, andresulting data is output at 240 frames per second.

In step S102, the camera control unit 11 or the camera control unit 111determines a recording unit 29 or a camera signal processing unit 25 or126 from which an image data stream is read on the basis of the setvalue of the output frame rate.

Specifically, in a case where data captured at a frame rate of 240frames per second in the imaging apparatus 1 or the imaging apparatus101 is divided into four segments, if the output frame rate is 60 framesper second, the four recording units 29 or one of the camera signalprocessing units 25 or 126 is selected as the target from which an imagedata stream is read. If the output frame rate is 120 frames per second,the four recording units 29 or two of the camera signal processing units25 or 126, which have discontinuous (shifted by two frames) frames ofdata, are selected as the target from which an image data stream isread. If the output frame rate is 240 frames per second, the fourrecording units 29 or all the camera signal processing units 25 or 126are selected as the target from which an image data stream is read.

In step S103, when video output is a process of outputting the datarecorded on the recording units 29, the camera control unit 11 or thecamera control unit 111 starts to read an image data stream from apredetermined recording unit 29 and to control a codec unit 28 toperform decoding processing.

In step S104, the camera control unit 11 or the camera control unit 111sets, for four frames, write start addresses of video image data to bewritten in the frame memory (not illustrated) of the video output unit26.

In step S105, the camera control unit 11 or the camera control unit 111initializes the value of the counter C for determining a codec unit 28or a camera signal processing unit 25 or 126 from which data to bewritten in the frame memory (not illustrated) of the video output unit26 is acquired.

Here, explanation will be given assuming that an image is captured at aframe rate of 240 frames per second, and four data segments aresubjected to frame combination, and resulting data is output at 240frames per second. Thus, it is assumed that the value of the counter Cis incremented in a range of 0 to 3. In contrast, in a case where framesthat constitute two-channel video image data of 60 frames per secondamong four-channel data segments are combined and a result is output at120 frames per second, the value of the counter C alternately takes 0and 2 or alternately takes 1 and 3. Additionally, in a case where videoimage data is output at 60 frames per second, the value of the counter Cis not incremented, and is constant in a range of 0 to 3.

In step S106, the camera control unit 11 or the camera control unit 111sets the read start address of video image data to be read for outputfrom the frame memory (not illustrated) of the video output unit 26 inthe beginning line of the frame to be read next among the four frames ofvideo image data that have already been recorded on the frame memory.

In step S107, the camera control unit 11 or the camera control unit 111determines whether or not the value of the counter C for determining acodec unit 28 or a camera signal processing unit 25 or 126 from whichdata to be written in the frame memory (not illustrated) is acquiredmeets the condition C=0.

In a case where it is determined in step S107 that the condition C=0holds, in step S108, the camera control unit 11 or the camera controlunit 111 writes the video image data supplied from the #1 codec unit28-1 or the #1 camera signal processing unit 25-1 or 126-1 in blocks,for example, for each line in the frame memory on the basis of the writeaddress of the first frame that is the temporally earliest among thefour frames, and the process proceeds to step S114.

In a case where it is determined in step S107 that the condition C=0does not hold, in step S109, the camera control unit 11 or the cameracontrol unit 111 determines whether or not the value of the counter Cfor determining a codec unit 28 or a camera signal processing unit 25 or126 from which data to be written in the frame memory (not illustrated)is acquired meets the condition C=1.

In a case where it is determined in step S109 that the condition C=1holds, in step S110, the camera control unit 11 or the camera controlunit 111 writes the video image data supplied from the #2 codec unit28-2 or the #2 camera signal processing unit 25-2 or 126-2 in blocks,for example, for each line in the frame memory on the basis of the writeaddress of the second frame that is the temporally second earliest amongthe four frames, and the process proceeds to step S114.

In a case where it is determined in step S109 that the condition C=1does not hold, in step S111, the camera control unit 11 or the cameracontrol unit 111 determines whether or not the value of the counter Cfor determining a codec unit 28 or a camera signal processing unit 25 or126 from which data to be written in the frame memory (not illustrated)is acquired meets the condition C=2.

In a case where it is determined in step S111 that the condition C=2holds, in step S112, the camera control unit 11 or the camera controlunit 111 writes the video image data supplied from the #3 codec unit28-3 or the #3 camera signal processing unit 25-3 or 126-3 in blocks,for example, for each line in the frame memory on the basis of the writeaddress of the third frame that is the temporally third earliest amongthe four frames, and the process proceeds to step S114.

In a case where it is determined in step S111 that the condition C=2does not hold, the condition C=3 holds. Thus, in step S113, the cameracontrol unit 11 or the camera control unit 111 writes the video imagedata supplied from the #4 codec unit 28-4 or the #4 camera signalprocessing unit 25-4 or 126-4 in blocks, for example, for each line inthe frame memory (not illustrated) of the video output unit 26 on thebasis of the write address of the fourth frame that is the temporallylast among the four frames, and the process proceeds to step S114.

After the completion of the processing of step S108, S110, S112, orS113, in step S114, the camera control unit 11 or the camera controlunit 111 increments the value of the counter C. Here, when the value ofthe counter C is 3, the counter C is initialized to 0.

In step S115, the camera control unit 11 or the camera control unit 111determines whether or not the writing of the four frames whose writestart addresses have been set in step S46 has been completed. In a casewhere it is determined in step S115 that the writing of the four frameshas been completed, the process returns to step S104, and the subsequentprocessing is repeated.

In a case where it is determined in step S115 that the writing of thefour frames has not been completed, in step S116, the camera controlunit 11 or the camera control unit 111 increments the write addresscorresponding to the frame for which writing has been performed in theprocessing of step S108, S110, S112, or S113.

In step S117, the camera control unit 11 or the camera control unit 111reads the imaging data recorded on the frame memory (not illustrated) ofthe video output unit 26 in blocks on the basis of the read address.

In step S118, the camera control unit 11 or the camera control unit 111determines whether or not the reading of one frame has been completed.In a case where it is determined in step S118 that the reading of oneframe has been completed, the process returns to step S106, and thesubsequent processing is repeated.

In a case where it is determined in step S118 that the reading of oneframe has not been completed, in step S119, the camera control unit 11or the camera control unit 111 increments the read address.

In step S120, the camera control unit 11 or the camera control unit 111determines whether or not completion of the video output has beencommanded. In a case where it is determined in step S120 that completionof the video output has not been commanded, the process returns to stepS107, and the subsequent processing is repeated. In a case where it isdetermined in step S120 that completion of the video output has beencommanded, the process ends.

With such a process as above, video image data of a frame rate that isone quarter the imaging frame rate, which has been divided into fourvideo image data segments, can be subjected to frame combination andoutput.

Note that, here, explanation has been given assuming that an imagecaptured at 240 frames per second is divided into four pieces which areprocessed as moving image data of 60 frames per second and recorded, andframe combination is performed, as necessary, during reproduction andoutput. However, it goes without saying that any other frame rate of acaptured image and any other number of segments may be used.

Specifically, for example, a moving image captured at 240 frames persecond may be divided into two or three segments, or a moving imagecaptured at 120 frames per second may be divided into two or threesegments. Additionally, a moving image captured at 200 frames per secondmay be divided into four segments, or a moving image captured at 100frames per second may be divided into two segments. Alternatively, amoving image captured at 96 frames per second may be divided into foursegments, or a moving image captured at 48 frames per second may bedivided into two segments.

At this time, when moving image data segments of each channel have aframe rate that is generally widely used for capturing a moving image,such as, for example, 60 frames per second, 50 frames per second, or 24frames per second, for example, a general-purpose product can be usedfor a circuit or the like necessary for signal processing or codec, andcost can be reduced.

Additionally, here, explanation has been given of a case where, by wayof example, the frame rates of segments of each channel are equal toeach other. However, it goes without saying that segments of eachchannel may have different frame rates.

Furthermore, in the foregoing explanation, explanation has been given ofa case where an image of HD resolution is captured. However, it goeswithout saying that the present invention can also be applied in a casewhere images of different resolutions are captured or in a case where animage is displayed in an interlaced format.

That is, an imaging apparatus to which the present invention is appliedincludes a solid-state imaging element capable of capturing an image ata frame rate that is N times a frame rate generally widely used forcapturing a moving image, and can divide the image into N segments inunits of frames to generate N channels of moving image data of a framerate that is 1/N the imaging frame rate. The respective channels ofmoving image data can be processed and recorded in parallel using Nparallel processing circuits. Then, an imaging apparatus to which thepresent invention is applied is capable of processing an imaging signalof a high resolution and a high frame rate or recording the imagingsignal for a long time without performing complex processing.

Furthermore, in an imaging apparatus to which the present invention isapplied, moving image data that is recorded in parallel is moving imagedata of a frame rate that is 1/N the imaging frame rate, that is, of anormal frame rate generally widely used for capturing a moving image.Moving image of the normal frame rate can be reproduced by reproducingone channel alone, moving image of a frame rate that is twice the normalframe rate can be reproduced by reproducing two channels, and an imageof a frame rate that is N time the normal frame rate can be reproducedby reproducing N channels.

Additionally, in the present invention, in order to execute processes inparallel, additionally, in order to record an image of a high resolutionand a high frame rate for a long time, imaging data is divided inminimum units of frames without dividing each frame of the imaging datainto a plurality of pieces, for example, for every slice or within apredetermined rectangular range. This allows the generated moving imagedata segments of each channel to be reproduced and displayed alone.Further, the number of frames to be combined is changed so as tofacilitate easy reproduction and output at different frame rates.Additionally, in a case where a moving image recorded at a frame ratelower than the frame rate of imaging data is reproduced, there may be noneed to perform codec for a channel that is not displayed.

By doing so, an imaging apparatus capable of recording an image of ahigh resolution and a high frame rate for a long time and capable ofoutputting reproduction data of a plurality of frame rates byrearranging frames, as necessary, using recorded moving image datasegments without performing complex processing can be provided.

Note that in a case where a moving image is displayed, in accordancewith an increase of the frame rate or field rate, the evaluation valueof jerkiness or motion blur, which is given by an observer who views adisplayed image, is improved. FIG. 15 illustrates an example of therelationship between the frame rate of a movie and the evaluation valuein a case where image quality was evaluated using five levels ascontrols for a plurality of users.

An evaluation of 4 or more out of five-level evaluation can be achievedon average for both jerkiness and motion blur at around 150 frames persecond, with the tendency that the evaluation value increases up to near250 frames per second, whereas the evaluation value does not increase somuch at a higher frame rate or field rate (see, for example, JapaneseUnexamined Patent Application Publication No. 2004-266808).

Many video resources that are presently widely used are of 50 frames persecond or 60 frames per second. Thus, an ideal frequency that takes theefficiency of video resources into consideration is a frequency that isan integer multiple of 50 or 60 frames per second, namely, 240 framesper second or 200 frames per second. When the output field rate has anyof the above values, an observer who views a displayed image does notobserve flicker or perceives substantially no jerkiness or motion blur,which is preferable. Additionally, since many video resources that arepresently widely used are of 50 frames per second or 60 frames persecond, when segments have a frame rate of 50 frames per second or 60frames per second, a general-purpose product can be used for a circuitthat performs signal processing or codec, and cost can be reduced.

The series of processes described above can be executed by hardware orsoftware. This software is installed from a recording medium into acomputer in which a program constituting this software is incorporatedin dedicated hardware or, for example, a general-purpose personalcomputer or the like that is capable of executing various functions byinstalling therein various programs. In this case, the processesdescribed above are executed by a personal computer 500 as illustratedin FIG. 16.

In FIG. 16, a CPU (Central Processing Unit) 501 executes variousprocesses in accordance with a program stored in a ROM (Read OnlyMemory) 502 or a program loaded from a storage unit 508 to a RAM (RandomAccess Memory) 503. The RAM 503 further stores, as appropriate, data andthe like necessary for the CPU 501 to execute various processes.

The CPU 501, the ROM 502, and the RAM 503 are connected to one anothervia an internal bus 504. An input/output interface 505 is also connectedto this internal bus 504.

The input/output interface 505 is connected to an input unit 506composed of a keyboard, a mouse, and the like, an output unit 507composed of a display composed of a CRT, an LCD, or the like, a speaker,and the like, the storage unit 508 configured by a hard disk and thelike, and a communication unit 509 configured by a modem, a terminaladapter, or the like. The communication unit 509 performs communicationprocessing via various networks including a telephone line and a CATV.

The input/output interface 505 is also connected to a drive 510, asnecessary, in which a removable medium 521 composed of a magnetic disk,an optical disk, a magneto-optical disk, a semiconductor memory, or thelike is placed, as appropriate. A computer program read therefrom isinstalled into the storage unit 508, as necessary.

In a case where the series of processes is executed by software, aprogram that constitutes this software is installed from a network or arecording medium.

This recording medium is not only configured by, as illustrated in FIG.16, a package medium composed of the removable medium 521 having theprogram recorded thereon, which is distributed separately from thecomputer in order to provide the program to a user, but is alsoconfigured by the ROM 502, a hard disk including the storage unit 508,or the like on which the program is recorded, which is provided to auser in a state of being incorporated in advance in the main body of anapparatus.

Note that in a case where the series of processes described above isexecuted by software, signal processing and codec may be executed by theCPU 501, or hardware components that perform signal processing and codecmay be prepared and the CPU 501 may execute a program for controllingthose hardware components (executing control basically similar to thecontrol executed by the camera control unit 11 or the camera controlunit 111).

Note that in this specification, steps describing a computer program aredesigned to include, as well as processes performed in times series inaccordance with the order described herein, processes executed inparallel or individually even through they are not necessarily processedin time series.

Note that embodiments of the present invention are not limited to theembodiment described above, and a variety of modifications can be madewithout departing from the scope of the present invention.

The invention claimed is:
 1. An imaging apparatus comprising: an imagingunit configured to obtain imaging data of a first rate; a data dividingunit configured to distribute the imaging data of the first rate, whichis captured by the imaging unit, in unit of frames and dividing theimaging data into N channels of moving image data of a second rate thatis a rate that is 1/N the first rate (where N is a positive integergreater than one); and N image processing units each implemented byprocessing circuitry and configured to process concurrently and inparallel the N channels of moving image data obtained by the datadividing unit, the processing of each of the N channels including colorspace conversion processing converting a color space of the moving imagedata.
 2. The imaging apparatus according to claim 1, wherein the colorspace conversion processing further converts the color space of themoving image data from a color space defined by RGB into a color spacedefined by luminance and chrominance.
 3. The imaging apparatus accordingto claim 1, further comprising: an output unit configured to output theN channels of moving image data processed by the image processing unit,wherein the output unit outputs only one channel of the N channels ofmoving image data or outputs a result obtained by performing framecombination on at least a portion of the N channels of moving image dataon the basis of a rate of moving image data to be output.
 4. The imagingapparatus according to claim 1, further comprising: a recording unitconfigured to record the N channels of moving image data processed bythe image processing unit, wherein the recording unit is furtherconfigured such that N recording units are provided or the recordingunit is divided into N areas, and respectively records the N channels ofmoving image data processed by the image processing unit.
 5. The imagingapparatus according to claim 1, wherein the second rate is 60 frames persecond.
 6. The imaging apparatus according to claim 1, wherein thesecond rate is 50 frames per second.
 7. The imaging apparatus accordingto claim 1, wherein the second rate is 24 frames per second.
 8. Theimaging apparatus according to claim 1, wherein the N channels are fourchannels.
 9. The imaging apparatus according to claim 1, wherein the Nchannels are two channels.
 10. The imaging apparatus according to claim1, wherein the first rate is 240 frames per second.
 11. The imagingapparatus according to claim 1, further comprising: a recording unitconfigured to record the N channels of moving image data processed bythe image processing unit, wherein the recording unit is furtherconfigured such that N recording units are provided or the recordingunit is divided into N areas, and respectively records the N channels ofmoving image data processed by the image processing unit.
 12. Theimaging apparatus according to claim 1, further comprising: a recordingunit configured to record the N channels of moving image data processedby the image processing unit; and an encoding unit configured to encodethe N channels of moving image data processed by the image processingunit, wherein the recording unit records the N channels of moving imagedata encoded by the encoding unit.
 13. The imaging apparatus accordingto claim 12, further comprising: a decoding unit configured to decodethe N channels of moving image data encoded by the encoding unit andrecorded by the recording unit; and an output unit configured to outputthe N channels of moving image data decoded by the decoding unit,wherein the decoding unit decodes only one channel of the N channels ofmoving image data or decodes at least a portion of the N channels ofmoving image data on the basis of a rate of moving image data to beoutput, and the output unit outputs the one channel of the N channels ofmoving image data, which is decoded by the decoding unit, or outputs aresult obtained by performing frame combination on at least a portion ofthe N channels of moving image data on the basis of a rate of movingimage data to be output.
 14. An imaging method for a portable imagingapparatus that captures moving image data, the imaging method comprisingthe steps of: performing, at the portable imaging apparatus, imaging ata first rate; dividing, at the portable imaging apparatus, capturedimaging data of the first rate in units of frames into N channels ofmoving image data of a second rate that is a rate that is 1/N the firstrate (where N is a positive integer greater than one); and concurrentlyprocessing the obtained N channels of moving image data using N parallelunits by processing the N channels of moving image data, the processingof each of the N channels including color space conversion processingconverting a color space of the moving image data.
 15. A non-transitorycomputer readable medium having stored thereon a program for causing acomputer to execute a process of capturing moving image data, theprogram causing the computer to execute a process comprising the stepsof: controlling a portable imaging apparatus to perform imaging at afirst rate; dividing, at the portable imaging apparatus, capturedimaging data of the first rate in units of frames into N channels ofmoving image data of a second rate that is a rate that is 1/N the firstrate (where N is a positive integer greater than one); and concurrentlyprocessing, at the portable imaging apparatus, the obtained N channelsof moving image data using N parallel units by processing the N channelsof moving image data, the processing of each of the N channels includingcolor space conversion processing converting a color space of the movingimage data.
 16. An imaging apparatus comprising: an imaging unitconfigured to obtain imaging data of a first rate; a data dividing unitconfigured to distribute the imaging data of the first rate, which iscaptured by the imaging unit, in units of frames and dividing theimaging data into N channels of moving image data of a second rate thatis a rate that is 1/N the first rate (where N is a positive integergreater than one); and N image processing units each implemented byprocessing circuitry and configured to process in parallel the Nchannels of moving image data obtained by the data dividing unit,wherein the second rate is 24, 50, or 60 frames per second.
 17. Animaging apparatus comprising: an imaging unit configured obtain imagingdata of a first rate; a data dividing unit configured to distribute theimaging data of the first rate, which is captured by the imaging unit,in units of frames and dividing the imaging data into N channels ofmoving image data of a second rate that is a rate that is 1/N the firstrate (where N is a positive integer greater than one); and N imageprocessing units each implemented by processing circuitry and configuredto process in parallel the N channels of moving image data obtained bythe data dividing unit, wherein the N channels are four channels.
 18. Animaging apparatus comprising: an imaging unit configured to obtainimaging data of a first rate; a data dividing unit configured todistribute the imaging data of the first rate, which is captured by theimaging unit, in units of frames and dividing the imaging data into Nchannels of moving image data of a second rate that is a rate that is1/N the first rate (where N is a positive integer greater than one); andN image processing units each implemented by processing circuitry andconfigured to process in parallel the N channels of moving image dataobtained by the data dividing unit, wherein the first rate is 240 framesper second.
 19. The imaging apparatus according to claim 11, wherein theimage processing unit includes at least one of a white balancecorrection unit, an RGB interpolation synchronization processing unit, amatrix processing unit, a gamma correction unit, and a color spaceconversion unit.
 20. The imaging apparatus according to claim 2, whereinthe color conversion processing converts pixel data based on an RGBcolor space into pixel data based on a YCbCr color space.
 21. Theimaging apparatus according to claim 16, wherein the processing of eachof the N channels includes color space conversion processing convertinga color space of the moving image data.
 22. The imaging apparatusaccording to claim 17, wherein the processing of each of the N channelsincludes color space conversion processing converting a color space ofthe moving image data.
 23. The imaging apparatus according to claim 18,wherein the processing of each of the N channels includes color spaceconversion processing converting a color space of the moving image data.